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ABSTRACT 

A value-centered approach to science, technology and society (STS) education illuminates 
the need of reflexive and relational learning through communication and public 
engagement. Visualization is a key to represent and compare mental models such as 
assumptions, background theories and value systems that tacitly shape our own 
understanding, interests and interactions. Yet conventional approaches including concept 
mapping and multi-criteria value elicitation methods often have little suggestion or 
implication as to how participants themselves can address and deliberate the 
incompatibility of their perceptions, preferences and perspectives. This study proposes Q 
workshop as a legitimate eliciting and deliberation technique that can be employed in 
pluralistic discourse, exploring systematic divergences of perspective by constructing the 
participant's self in a formative, emergent and contingent manner. For this it introduces 
Q mapping as a novel visualization tool for the hybridity of qualitative and quantitative 
methods derived from Q methodology. Q mapping is a two-factor solution that transforms 
the similarities in participants’ individual Q scores into distances represented in two- 
dimensional space, for the sake of illustrating the relative positioning and partitioning of 
perspectives in a schematic figure. A case study on STS education for postgraduate 
students demonstrates that Q workshop can play a heuristic and abductive role in 
providing independent illumination of distinguishable perspectives and facilitating 
individual and collective learning among participants, suggesting a schematic two- 
dimensional basis for resolving the key differences. 
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Introduction: Representation of Mental Models 

Over the last forty years science education research has recognized that 
science should be placed in the context of how it affects technology and society. A 
recent movement of science, technology and society (STS) education deals 
sensibly with moral and ethical reasoning of socioscientific issues (Zeidler, 
Walker, Ackett, & Simmons, 2002; Zeidler, 2003; Sadler, 2004; Donnelly, 2004; 
Zeidler, Sadler, Simmons, & Howes, 2005). A value-centered approach (Pedretti 
& Nazir, 2011) reveals that learning socioscientific issues encourages people to 
actively reflect and examine the relevant connections between science and their 
own values and perspectives (Allchin, 1999; Driver, Newton, & Osborne, 2000; 
Kolsto, 2001, 2006; Bell & Lederman, 2003; Sadler, 2004; Sadler & Donnelly, 
2006; Sadler, Barab & Scott, 2007; Zeidler, Sadler, Applebaum, & Callahan, 
2009). Science communication studies have also begun to place more emphasis 
on understanding the multifaceted relationship between science and the variety 
of values, as well as the importance of reflective thinking and deliberation 
(Nisbet & Scheufele, 2009; Dietz, 2013). This illuminates the need of reflexive 
and relational learning through communication and engagement in science 
education by moving away from such labels as “experts” and “lay citizens” 
(Cuppen, 2012a; Chilvers & Kearnes, 2016). Reflexive learning concerns insights 
into the assumptions, background theories and value systems that tacitly shape 
our own understanding, interests and interactions (Felt & Wynne, 2007; 
Chilvers, 2013; VoB & Kemp, 2006) on the basis of double-loop learning (Argyris 
& Schon, 1978). Relational learning involves “learning about the independent 
integrity of others” (Felt & Wynne, 2007, p. 63). 

Values are created and developed by frames (Nisbet, 2009), worldviews 
(Cobern, 1996; Lacey, 2009) or mental models (Bruine de Bruin & Bostrom, 
2013) through which people interpret new information. One difficult challenge is 
how to represent mental models for communication, engagement and learning. A 
mental model is an internal conceptual representation that corresponds to the 
representation of the external structure (Gentner & Stevens, 1983). According to 
a less instructional but more theoretical and dynamic approach, a mental model 
continues to be enlarged and improved as new information is incorporated into it 
(Doyle & Ford, 1998; Greca & Moreira, 2000; Jones, Ross, Lynam, Perez, & 
Leitch, 2011). Representation of mental models can be accomplished by the 
externalization of information through visual models (Mnguni, 2014) and the 
form of this visualization affects the structure of mental models acquired during 
learning (Ramadas, 2009; Schnotz & Kiirschner, 2008). Externalization of 
mental models is also a key in cooperative learning (Scardamalia & Bereiter, 
1987). Cooperative learning is an instructional technique whereby people work 
together in small groups to improve their own and each other’s learning (Cooper, 
1995; Johnson & Johnson, 1999; Bowen, 2000). It has great effects on 
participants’ learning when groups are focused on the individual learning of 
their members (Slavin, 1996). Reflexive learning then occurs at the collective 
level through shared mental models (Kim, 1993; Espejo, 1996), visualization of 
which can facilitate communication and consensus between participants (Swaab, 
Postmes, Neijens, Kiers, & Dumay, 2002). 
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Among the most common methods to represent mental models (Carley & 
Palmquist, 1992; Jonassen & Cho, 2008; Al-Diban & Ifenthaler, 2011) is concept 
or cognitive mapping (Novak, 1990). It is now widely applied as a graphical 
node-arc representation of the relationships among a collection of concepts that 
reflects some aspect of subjectivity, and is thought to support understanding, 
learning and decision-making (Lumpkin, 1999; Kolkman, Kok, & van der Veen, 
2005; Hay, Kinchin, & Lygo-Baker, 2008; Daley & Torre, 2010). There are 
various ways to articulate mental models at the group level, with a link to the 
idea of a shared understanding (Kim, 2009; Scholz, Austermann, Kaldrack, & 
Pahl-Wostl, 2015). Whereas most methods focus on mental models and their 
changes within subjects, only a few compare mental models between subjects 
(Schaffernicht and Groesser, 2011). Cluster analysis can categorize and compare 
mental models as a metric (Mathieu, Heffner, Goodwin, Cannon-Bowers, & 
Salas, 2005), but it requires a great number of samples (large-N) to confirm its 
validity, which reduces its applicability for participatory practice in terms of 
sample size and representativeness. More applicable may be multiple criteria 
value elicitation methods for stakeholder analysis (Hermans & Thissen, 2009), 
including the analytic hierarchy process (AHP) (Lu, Lian & Lien, 2015) and 
multi-criteria mapping (MCM) (Stirling & Mayer, 2001; Burgess et al., 2007). 
These group-based practices aggregate different mental models at the collective 
level and/or compare them in an illustrative way but do not make sensible 
suggestions on how to regroup them beyond preset categorization of 
stakeholders. In addition, time-consuming nature of these approaches is 
seriously problematic for one-shot participatory practice to facilitate immediate 
communication and learning. To address such challenge, this study 
demonstrates ‘Q workshop’ applying Q methodology as a public engagement 
exercise to think about and learn different mental models. 

Q Methodology 

What is currently referred to as Q methodology was created over 80 years 
ago by William Stephenson. The methodology was named Q methodology to 
make a contrast with the conventional approach to the use of multivariate factor 
analysis in the study of psychological processes, which he labeled ‘R 
methodology’. Among other stakeholder analyses, there are a variety of 
advantages to Q methodology. First, it excels at quantitatively identifying 
different frames shared by specific groups of stakeholders in participation, even 
when these are implicit. In other words, Q methodology deliberately pursues 
‘social representations’ that are linked with a special way of acquiring and 
communicating knowledge (Watts & Stenner, 2005; Moscovici, 1981). Second, 
notwithstanding the prominence of quantitative procedure, Q methodology 
effectively combines the strengths of both qualitative and quantitative research 
traditions since it is designed to examine an individual’s subjective experience 
that is typically passed over by other quantitative procedures (Brown, 1996; 
Stenner & Stainton Rogers, 2004). Third, Q methodology does not require any 
presupposition about the structure of research subject and participants, where 
other methodologies might find it difficult to avoid preconceived demographic 
and occupational notions. Fourth, it is more feasible because of the relatively few 


6280 


G. YOSHIZAWA ET AL. 


06 > 

human, financial and time resource requirements and because it permits a 
considerable degree of flexibility in choosing subject items and participants. 
Stephenson’s original ideas have recently gained in prominence in academia, 
where a number of researchers have pursued the implications and applicability 
of his ideas in communication (Giannoulis, Botetzagias & Skanavis, 2010), policy 
studies (Addams & Proops, 2000; Ockwell, 2008; Cairns, 2012; Cairns & Stirling, 
2014), stakeholder engagement (Sleenhoff, Cuppen & Osseweijer, 2015; Cotton 
& Mahroos-Alsaiari, 2015) and other related areas (Dziopa & Ahern, 2011). 
Among a number of elicitation methods for identifying individual mental 
models, some observe that only Q methodology supports an analytically 
grounded grouping of mental models and provides the relative distribution of 
different mental models (Raadgever, Mostert, & Van de Giesen, 2008; Day, 
2008). A couple of policy studies have so far attempted to facilitate mutual 
understanding and communication between participants using Q methodology 
(Cuppen, Breukers, Hisschemoller, & Bergsma, 2010; Cuppen, 2012b). There are 
relatively few studies in education research. All of these studies apply Q with an 
aim to systematically engage with and interpret participants’ values, opinions 
and attitudes in relation to learning but not to facilitate their learning per se 
(Cothran & Ennis, 1998; Lecouteur & Delfabbro, 2001; Cross, 2005; Barker, 
2008; Ramlo, McConnell, Duan, & Moore, 2008; Ramlo, 2012; Chen, Chen & 
Chen, 2015). 

In Q methodology, a set of participants (P set) are usually selected as 
‘discursive representatives’ (Dryzek & Niemeyer, 2008) in order to ensure that 
there is enough diversity among their perspectives in the group, and not 
necessarily to provide an accurate mirror of the population (Watts & Stenner, 
2012, pp. 70-71). Individual participants conduct a rank-order procedure of a set 
of samples (Q samples). Samples are usually collected from a random collection 
of self-referable statements about something conceived for any situation or 
context. This ‘concourse’ of statements (Stephenson, 1978) can be 
operationalized as the complete population of statements made in or around the 
area at hand (Dryzek, 1990). The Q set is normally a set of statements that 
serve as a reasonable representation of the concourse. Given a risk of 
‘unstructured’ samples that are under- or over-sampled and inadvertently 
introduce a bias of some kind into the final Q sample, theoretical structuring is a 
more systematic procedure often by using a cell structure as a heuristic device 
for selecting statements (McKeown & Thomas, 1988). Individual participants 
are then asked to sort Q samples according to a sorting grid in the form of a 
forced quasi-normal distribution (Q sort). In terms of the range of distribution 
(i.e. the width of the Q sort scale), the larger the number of statements, the 
wider the range of available scores should be. Taking a range of +5 to -5 as 
generally employed for the number of statements from 40 to 60 (Brown, 1980, p. 
200), less than 40 statements for a Q set is often allowed to conduct Q sorting. 
The general shape of the distribution should vary in complexity, as should the 
uncertainty of the issues and the participants’ knowledge and expertise. 

After Q sorts are collected and correlated, factor analysis is performed to 
identify the underlying structure of relationships between the Q sorts. In doing 
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so, firstly, it is necessary to decide the method of extracting ‘factors’, which 
represent the underlying dimensions that account for the original set of observed 
variables (i.e. Q sorts). The correlation of each variable and the factor is called 
‘factor loading’ and this is the means of interpreting the role each variable plays 
in defining each factor. Accordingly, participants associated with higher loadings 
for a factor are the representatives of the factor. An important tool to have a 
more meaningful pattern of factor loadings is factor rotation, which generates 
heavy loadings of each Q sort on one factor and trivial loadings on the other. 
This is achieved by a process called rotation that can be conducted either 
according to mathematical criteria (e.g. varimax rotation) or by a visual 
procedure (i.e. judgmental rotation, or ‘hand rotation’). Both methods are on a 
par and it is reasonable to employ one method according to the aim of the 
research (Brown, 1980, p. 238). 

In participatory practice, once the factors are identified, the participants 
are broken down into groups according to their Q methodology factor. In order to 
explain and understand the different perspectives, each group usually begins 
with an analysis of the extreme poles of the Q factor array and then moves 
toward more neutral scoring statements. Through consideration of within-factor 
variation and comparisons with other factor arrays, it builds up a narrative, 
summarizing account of the factor (McKenzie, Braswell, Jelsma, & Naidoo, 
2011 ). 

Q Workshop 

A considerable number of studies have demonstrated the utility of Q 
methodology for specific research aims, the support of stakeholder consensus, 
decision making and public policy (Durning & Brown, 2006; Brown, Durning & 
Selden, 2008), it being centered on the analysis of the resemblance between 
persons with regard to measures of agreement (Dijkstra & van Eijnatten, 2009). 
In participatory research, Q methodology has also been regarded as a practical 
tool to select participants for a stakeholder dialogue (Cuppen et al., 2010), as 
well as to identify changes in perspectives before and after the dialogue 
(Cuppen, 2012a). Yet a problem arises when researchers downplay elicitation of 
the partitioning and relative positioning of perspectives for reflexive and 
relational learning. Conventional applications of Q methodology remain a static 
and passive presentation of different (and possibly contentious) perspectives, 
which often have little suggestion or implication as to how participants 
themselves can address the incompatibility of these perspectives to communicate 
with and learn from each other. It is curious that while Stephenson (1967, 1969, 
1978, 1986), among others, has insisted on the subjective and communicative 
functions of Q methodology, thus far little effort has been put into developing 
methods to improve participants’ individual understanding and mutual 
communication and learning through Q methodology. As a remedy, Q workshop 
deliberately takes different steps in sampling participants and statements from 
standard Q applications and introduces a novel visualization tool in combination 
with R factor analysis. 


P set 
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Strategic sampling as a standard way to recruit participants for Q 
methodological research may not be recommendable where researchers 
emphasize and attempt to facilitate interaction between participants in order to 
identify the differences in their perspectives, views and thoughts regarding 
social interaction including mutual understanding and learning, problem 
identification, issue framing, agenda setting, consensus building, and decision¬ 
making. As examined later, Q workshop can function more fully in heuristic and 
action-oriented participatory research, which necessarily entails a more 
opportunistic approach to participant sampling. 

Qset 

It is significant that in Q methodology participants propose, deliberate and 
arrange statements themselves, which goes beyond simple random sampling 
(and large sample size) or unstructured sampling. Instead of the conventional 
division between structured and unstructured sampling techniques, Yuichiro 
Takahashi (personal communication, January 27 & February 25, 2015) proposes 
three types of sampling for Q sets: theoretical, exploratory and deliberative. 
First, in ‘theoretical sampling’, which can often be observed in psychology, the 
investigator uses a set of theoretically established (and empirically confirmed) 
statements. Second, ‘exploratory sampling’ refers to the process in which the 
investigator assumes a set of statements based on literature review and expert 
interview and confirms the Q set reflecting individual participants’ feedback on 
the relevance of statements. In a Q study, for example, researchers impose a 
Fisherian grid on statements selected by participants from the existing study 
(McKenzie et al., 2011). 

Third, in the case of ‘deliberative sampling’, all the participants propose and 
deliberate statements at a gathering and complete a Q set. In the course of 
deliberation, which is usually facilitated by the investigator or a professional 
facilitator, some statements are separated, shortened, abstracted or 
incorporated together into single short statements to improve the clarity if there 
are any unclear, ambiguous, double barreled, or similar statements in meaning. 
These participatory approaches can avoid the risk of unstructured sampling that 
some opinions will be under- or oversampled and constructing a Q set that 
participants do not understand or know what the statements mean. They can 
also avoid the risk of structured sampling framed by the investigator’s own 
values and interests. Deliberative sampling enables participants to digest the 
statements more quickly and smoothly and consequently shorten the time of Q 
sorting. Furthermore, such real-time collective sample collecting and editing 
through interaction and collaboration may provide the participants with a sense 
of ownership of all the statements and a sense of commitment to the completion 
of Q sorting and the analysis result. For instance, some Q studies conducted 
workshops to generate statements that express participants’ perspectives 
(Mattson, Byrd, Rutherford, Brown, & Clark, 2006; Mattson, Clark, Byrd, 
Brown, & Robinson, 2011). In the sense that meaning is organized in relation to 
different configurations of the statements of a concourse (Stephenson, 1978), the 
shared process of constructing a concourse makes finalized statements 
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communicable with all the participants as ‘common knowledge’ (Stephenson, 
1982) where the connotation of a statement is at risk of being strongly 
influenced by the proposed participant’s own voice and context. 

Qsort 

In deliberative sampling, which Q workshop normally adopts, participants 
are more familiar with individual statements because they can have common 
knowledge on the statements when proposed by other participants and 
confirmed by everyone at a workshop. It is inadvisable to take a much flatter 
distribution for the Q sorting grid since there is little information on 
participants’ knowledge and perceptions in advance of the Q sorting. As the 
bottom line it is safe in most cases to take a shape more similar to the normal 
distribution. 

Q. factor analysis 

In Q workshop as a real-time participatory appraisal, varimax rotation 
considerably saves time and effort to have analytical results with the help of 
dedicated computer software like PQMethod (Schmolck, 2011). 

R factor analysis 

In order to make the case visually illustrative for understanding, R factor 
analysis of Q sort data is applied as another quantitative analytical method. The 
distinction of R analysis and Q analysis is reviewed in great detail elsewhere 
(Burt & Stephenson, 1939; Stephenson, 1953; Brown, 1980; Durning, 1999) and 
entering this ongoing debate (Danielson, 2009; Brown, 2009; Kampen & Tamas, 
2014; Brown, Danielson & van Excel, 2015; Tamas & Kampen, 2015) is clearly 
beyond the scope of this article, so suffice it to say here that R analysis is 
primarily concerned with attributes, whereas Q analysis is primarily concerned 
with objects. Unlike in the cases using Likert-type scale data (Martin & 
Steelman, 2004; Thompson et al., 2012), a fundamental problem of our 
combinative method is that Q sort scores are not independent each other and 
thereby R analysis is strongly affected by Q methodology. As prominent 
advocates of Q repeatedly warn when using Q and R in parallel (Stephenson, 
1953, p. 15; Brown, Danielson, & van Exel, 2015), the use of R analysis should 
be regarded as a supplementary representation of the Q sort analysis results for 
externalization of mental models. 

This study takes principal component analysis as a basic model to obtain 
factor solutions since our primary concern is extracting two factors that account 
for the maximum portion of the total variance. The factor rotation method after 
the extraction is the varimax method, which seems to give a clearer separation 
of the factors than other methods and which has proved very successful as an 
analytic approach to obtaining an orthogonal rotation of factors. Factor loadings 
by R factor analysis indicate the degree of correspondence between the variable 
and the factor, with higher loadings making the variable representative of the 
factor. It is generally applicable to define factor loadings of more than ±0.40 as 
practically important when the sample size is 100 or larger (Hair, Anderson, 
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Tatham, & Black, 1998, p. 111). A more practical criterion we usually apply is to 
adopt statements with the highest 5 and lowest 5 loadings as the variable 
representative of the factor. Thus we have 4 sets of 5 statements for higher and 
lower factor loadings for Factor 1 and 2. The investigator then names each set by 
examining what these 5 statements represent or imply for the theme of Q sort. 
This is evidently subjective judgment and requires professional expertise, which 
can be biased by the investigator’s own interest and concern. Mathematically 
speaking, it does not make sense to put names to both higher and lower factor 
loadings for the same factor because a set of statements for lower loadings 
should mean exactly the opposite to that for higher loadings. Thus, if we label 
“X” for a set of higher statements for a factor, a set of lower statements for the 
factor should be labeled “non-X”. Statistically, however, the two-factor solution is 
the result of extracting statements into only two factors. This means that one 
factor can encompass many different sets of ideas, meanings and perspectives, 
which can allow us to have two labels at each end for better interpretation of the 
factor. 

Unlike with Q methodology, the sample size seriously matters in the R 
factor analysis. As a general rule, the minimum sample size will have at least 
five times as many observations as there are variables to be analyzed, and a 
more acceptable size would have a ten-to-one ratio (Hair et al., 1998, p. 99). 
However, an empirical test shows that even the ratios of 1.3 to 1 may produce 
recognizable factor solutions (Arrindell & van der Ende, 1985). Brouwer (1992- 
93) demonstrates the feasibility of using the same data for both Q and R 
analysis from a case of 80 variables with 54 observations. On balance then, R 
factor analysis at Q workshop is conducted for the sake of illustrating the 
relative positioning and partitioning of perspective in a schematic figure, not for 
determining a definitive and statistically rigorous picture of the configurations 
of data. 

Q. mapping 

The two-factor solution provided by R analysis transforms the similarities 
in participants’ individual Q scores into distances represented in two- 
dimensional space as in Figure 1. The horizontal axis and the vertical axis in the 
above figure represent the normalized factor scores of Factor 1 and Factor 2, 
respectively. In this example of a stakeholder analysis, on the horizontal axis 
the view spreads from ‘substantive’ in which people observe that the social 
appraisal of nuclear technology is useful in positive terms, to ‘symbolic’ in which 
they are doubtful about this, seeing the utility more in political terms. The view 
on the vertical axis spreads from ‘democratic’ (under which it is seen that the 
appraisal involves a wide range of stakeholders and the public) to ‘technocratic’ 
(under which it is seen that the appraisal involves only a limited range of 
experts) (Yoshizawa, 2007, p. 239). The factor scores are computed by the 
regression method, which works by multiplying the factor loadings by the 
inverse of the original correlation matrix. Conceptually, the factor score 
represents the degree to which each individual score obtained through Q sorting 
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rates high on statements that have high loadings on a factor. In Figure 1, each 
data point corresponds to the participant identification number. 



Perspectives from the Q factor analysis in Figure 1 are shown like a Venn 
diagram. Each closed curve represents an aggregate perspective. A conjunction 
between two closed curves means that participants therein share both 
perspectives. In the two-dimensional schematic figure, the Q result is overlaid 
on the R result. Each data point corresponds to an individual participant in the 
Q study. The position of the individual reflects the result of R factor analysis, 
whereas the group of individuals reflects the results of Q methodology. This 
makes Q mapping more analytically grounded than other methods that use 
pictorial representations of factors (Niemeyer, Petts & Hobson, 2005) or 
conceptual space diagrams on which various inter-factor relationships and 
differences in perspective are graphically but hypothetically represented 
(Stenner, Dancey & Watts, 2000; Watts & Stenner, 2005, 2012). Q map looks 
similar to the graphical representation of cluster analysis. Compared to cluster 
analysis, Q factor analysis is a more promising method to classify participants 
(Morf, Miller & Syrotuik, 1976), but the number of categories formed is large 
and the category profiles are less mutually exclusive with considerable overlaps 
(Chatterji & Mukerjee, 1986). As Figure 1 shows, Q mapping can take this 
disadvantage as an opportunity in disguise by depicting inter-factor 
relationships and illustrating overlaps of the factors. Even within a group, a 
participant can understand his or her own positioning in the group. 

Factor interpretation 

The partitioning and relative positioning of perspectives in a schematic way 
illustrated by Q mapping helps us to understand the result of Q methodology in 
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a visually intuitive manner. It further facilitates participants’ understanding of 
the self and others through the deliberation of Q mapping results. Although it is 
possible and reasonable in some cases for the investigator to collect Q sorts from 
the participants respectively through a face-to-face interview and draw a Q map 
afterwards (e.g. Yoshizawa, 2007), Q mapping is probably a more promising tool 
for settings where all the participants gather and conduct Q sorts individually in 
a workshop style. As described in more detail later, the latter case gives an 
opportunity for the participants to look at the configuration of different 
perspectives on the resulting Q map and reflexively learn their own position in 
the whole configuration. Through this process they will likely gain a better 
understanding of how to communicate and interact with other participants, as 
well as with non-participating stakeholders and citizens. 

A Case: STS Education 

The case study was conducted in a workshop on science and technology 
policy for postgraduate students in the Graduate University for Advanced 
Studies (known as ‘Sokendai’) in Japan. Sokendai is a university affiliated with 
research institutes and museums (IURIs: Inter-University Research Institutes) 
administered by the Ministry of Education, Culture, Sports, Science and 
Technology (MEXT). As Sokendai has no special facility in the IURIs, students 
are mentored at Sokendai both by the faculty of Sokendai and of the IURIs. 
Their disciplines significantly vary and their affiliated IURIs are located all over 
the world. Thus, for them and Sokendai itself, it is crucially important to gather 
postgraduate students at one place and facilitate interdisciplinary knowledge 
exchange and mutual learning. The three-hour workshop was conducted on July 
15, 2012 as a part of the annual two-day introductory event for postgraduate 
students who want to be researchers. 

P set 

The workshop participants as Q sorters consisted of 12 postgraduate 
students and 4 academic researchers mostly based in Sokendai. Their 
participation in the workshop was voluntary, which meant that investigators did 
not strategically select the participants. The P set may have been more biased 
than random sampling in the sense that volunteers are presumably keen to 
tackle with the issue at hand. However, this exercise itself did not aim to pursue 
academic rigor or explore social implications. The number of Q sorters might not 
be sufficient to conduct proper Q analysis and get a neat set of factors, but with 
hindsight the number was sufficient, as concerned participants were more 
familiar with thinking about the given issue and therefore had clearer and more 
consistent views on the topic than other postgraduate students who were not 
necessarily interested in the issue. 

Qset 

Statements were collected by deliberative sampling. Each participant was 
asked to think up 4 or 5 one-sentence statements replying to a question “who 
should do what for social contributions to research?” and write down these on 
sticky note. A facilitator asked the participants to devise short statements that 
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specify who should do what. Then, they presented their statements in turn. 
When one’s presented statement was the same as or similar to another’s 
statement in hand, these were shared with everybody at the same time. Later 
presenters only showed unique statements that had not already been presented. 
Statements were input into a computer and simultaneously projected onto a 
screen. The facilitator checked all the statements with the participants and 
similar statements were consolidated. The final set of statements was once again 
shared with everybody. This interactive process lasted about 40 minutes and 
ended up with 37 synoptic statements. 

Qsort 

This case analysis followed the standard Q sorting procedure and adopted 
37 sample statements on social contributions of research. Immediately after the 
Q set was finalized, each of the 37 statements was printed out on blank business 
cards numbered from 1 to 37. Although random numbering is preferable in order 
to disconnect any relations between statement and number, the numbering was 
more or less in the order of participants’ presentation and/or facilitator’s 
arrangement for the sake of convenience. Participants were asked to thoroughly 
shuffle and rank cards according to the sorting grid illustrated in Figure 2. 
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Figure 2. Q sorting grid for 37 statements 

They were then asked to transcribe their sort onto the sorting grid depicted 
in a response sheet. The sheet also listed the following questions. 

• Why did you choose the statements you most agree with? 

• Why did you choose the statements you most disagree with? 

• Are there any curious statements or any other issues in the sorting? 

The answers were to be used by the investigator for the reflection and the 
feedback during and after the workshop exercise. 

Q. factor analysis 
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This case study employed the centroid method to extract factors and 
varimax rotation to have a meaningful pattern of factor loadings. A Q factor 
analysis for the 16 participants led to the factor solution shown in Table 1. 
Factor loadings in excess of ±0.42 are statistically significant at the 0.01 level. 
However, in order to facilitate participants’ mutual learning by categorizing 
each of them into one group, this study used flagging factors automatically 
calculated and indicated in PQMethod. Flagged loadings are represented in bold 
text and shaded grey for ease of identification in Table 1. Participants are 
numbered according to the significant factor loadings for each factor. 

Table 1. Participants' factor loadings 


Factor Loadings 


ID 

Sex 

Status 

Discipline 

Factor A 

Factor B 

Factor C 

1 

M 

Professor 

Science and technology studies 

0.68 

0.08 


-0.31 

2 

F 

Student 

Physiological science 

0.62 

0.36 


0.36 

3 

F 

Student 

Food and nutritional science 

0.62 

-0.08 


0.22 

4 

F 

Postdoc 

Science communication 

0.60 

0.04 


-0.00 

5 

M 

Postdoc 

Science and technology studies 

0.59 

0.04 


0.19 

6 

M 

Student 

Fusion science 

0.51 

0.09 


0.20 

7 

F 

Student 

Evolutionary studies 

0.42 

0.01 


-0.04 

8 

M 

Student 

Applied biological chemistry 

0.42 

-0.22 


0.20 

9 

M 

Student 

Evolutionary studies 

0.46 

0.77 


-0.04 

10 

M 

Professor 

Quantum chromodynamics 

0.15 

0.52 


-0.04 

11 

M 

Student 

Molecular genetics 

0.02 

0.40 


0.10 

12 

M 

Student 

School education 

-0.14 

0.39 


0.12 

13 

M 

Student 

Molecular genetics 

-0.10 

-0.23 


0.59 

14 

M 

Student 

Space energy engineering 

0.13 

0.25 


0.44 

15 

M 

Student 

Materials structure science 

0.22 

0.16 


0.24 

16 

M 

Student 

Higher education 

0.15 

-0.32 


0.14 


R factor analysis 

The resulting Q sorts were independently factor analyzed. Factor 
extraction, rotation and factor score computation were processed by the 
dedicated Excel macro program (Aoki, 2006). Statements with the highest 5 and 
lowest 5 loadings are regarded as the variable representative of the factor. In 
the end, all statements, except one (#28 for Factor 2, see Table 4), had factor 
loadings of more than ±0.40. These important functions contain some ‘noise’, but 
this may well characterize each factor in the following interpretation. 


Table 2. Statements with the 5 highest factor loadings for Factor 1 

# Statement Loading 

15 Research institutes should hold open lectures .642 

33 Researchers should pass down the fun of science for future generations .618 

17 Universities should contribute to society with research findings .596 

22 Students should study their fields of interest in the first place .511 
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1 Mass media should cover even minor discoveries and post-doc research .498 


The statements with higher loadings for Factor 1 referred to as “working 
with the outside world”, include many that universities, research institutes and 
researchers themselves need to communicate the fun and interest of science and 
research findings with society. At the same time, mass media are expected to 
pick up small findings, which students should study carefully (Table 2). 


Table 3. Statements with the 5 lowest factor loadings for Factor 1 

# Statement Loading 

19 The government should foster research administrators and networkers -.788 

30 Universities should foster research administrators and networkers -.657 

9 Off-campus networkers should promote research -.639 

34 Graduate schools should provide university-wide career education -.577 

24 University should teach undergraduate students about social action -.542 


Lower loadings for Factor 1, labeled as “strengthening inside the academia”, 
are associated with statements that prioritize research administrators and 
networkers to researchers themselves and therefore need to support such 
human resources by government institutions and under- and post-graduate 
education (Table 3). 

Table 4. Statements with the 5 highest factor loadings for Factor 2 

# Statement Loading 

6 Companies should appreciate research career in recruiting .701 

31 Public sectors should assure basic management expense at the same level as .700 

before the transformation of incorporated administrative institutions 
10 University research administrators should generate ideas for research .593 

application which researchers may not be able to think of on their own 

1 Mass media should cover even minor discoveries and post-doc research .516 

28 The government should enhance international competitiveness by supporting .391 

gifted young researchers 

The statements with higher factor loadings for Factor 2 concern the support 
of researchers by the government, public sectors, companies, mass media and 
university research administrators, and can be termed as “social supports” 
(Table 4). 

Table 5. Statements with the 5 lowest factor loadings for Factor 2 

# Statement Loading 

35 Individual scientists should disseminate current issues and limitations of their -.765 

discipline 

8 Researchers should think about how their research area can address social -.717 

issues 

3 Mass media should pay attention to social problems in a skillful manner -.636 

16 Universities should offer classes to the general public -.584 

37 Researchers should communicate with researchers working in different -.556 
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The statements with lower factor loadings for Factor 2 are named as “self- 
help researchers”. They ask researchers to think about research issues and 
possible social contributions and communicate with other researchers in a 
transdisciplinary way. In order for researchers to look outward, it is necessary 
that mass media to cover social problems and universities become more open to 
the society (Table 5). 

Q. mapping 

The above two-factor solution transforms the similarities in participants’ 
individual Q scores into distances represented in two-dimensional space. In the 
workshop, data points were first displayed without an index (Figure 3). The 
workshop facilitator asked participants to guess which data point is his or hers 
own from a data plot with 4 labels at both extremes of the horizontal axis 
(“working with the outside world” and “strengthening inside the academia” for 
Factor 1) and the vertical axis (“social supports” and “self-help researchers” for 
Factor 2). Each participant put his or her name at a guessed data point. 
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Figure 3. Q map (in process) 

Then, the facilitator showed the figure with the name assigned to each data 
point (in Figure 4, the names are replaced with serial numbers for the sake of 
anonymity). The result illustrates that some guessed correctly while others 
guessed incorrectly. This purpose is not to evaluate the correctness of their 
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conjectures but rather to attract attention to and increase awareness of the 
relative positioning of their mental models. All of the closed curves are manually 
drawn in the shape of ovals for clarity of presentation and this does not reflect 
the outcome of statistical calculation. 



Figure 4. Q map (completed) 

Factor interpretation 

In the workshop, participants were divided into three groups according to 
the three-factor solution of Q methodology. Each group discussed what is 
characteristic of the group by looking at the results of Q methodology and the 
positioning and partitioning of the group on the Q map as well as the response 
sheets. The facilitator then asked each group to determine the group name and 
to give short presentation on the discussion results in turn. This process 
developed a sense of intimacy between group members and their collective 
identity. Positioning of each perspective in this schematic field is explained in 
the following. 
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8 participants (including 3 researchers) have significant positive loadings 
for Factor A and form the largest group. Table 6 shows statements significant at 
the 0.05 level (asterisk indicates at the 0.01 level) for Factor A and their factor 
scores with each factor. Factor scores are the ranked items converted back into 
the original Q sort distribution in order to create a factor array for each 
significant factor. The group members primarily support statements like 
“researchers should think about how their research area can address social 
issues” [#8] and “researchers should locate their own research in the whole 
research field and imagine the application of the research” [#36]. Through the 
discussion they became aware of how to link their own technology and 
knowledge with societal problem solving and their position among various 
research areas. On the other hand, in favor of “strengthening inside the 
academia”, they did not prioritize statements like that researchers are to be 
supported and research environment is to be arranged by society. Therefore they 
called themselves “seekers” in the sense that they did not expect much support 
from others. 


Table 6. Factor A - "Seekers" group 




Factor Scores 


# 

Statement 

A 

B 

C 

8 

Researchers should think about how their research area can address social 

issues 

5* 

3 

0 

36 

Researchers should locate their own research in the whole research field 
and imagine the application of the research 

4 * 

2 

3 

26 

Researchers should share common interests with general public 

3* 

-1 

-2 

19 

The government should foster research administrators and networkers 

3 

-1 

0 

24 

University should teach undergraduate students about social action 

2 * 

-4 

-1 

3 

Mass media should pay attention to social problems in a skillful manner 

2 

-1 

-1 

22 

Students should study their fields of interest in the first place 

2 

3 

-3 

29 

The government should provide student aid 

0 * 

4 

3 

25 

Scientists should communicate not only the fun of their own expertise but 
also that of science in general 

0 

3 

-2 

10 

University research administrators should generate ideas for research 
application which researchers may not be able to think of on their own 

0 * 

-2 

4 

5 

Companies should exchange more information with universities and 
research institutes 

0 * 

-3 

3 

16 

Universities should offer classes to the general public 

- 2 * 

2 

-4 

17 

Universities should provide the society with research finding 

- 2 * 

2 

2 

15 

Research institutes should hold open lectures 

-3* 

1 

1 

1 

Mass media should cover even minor discoveries and post-doc research 

-5* 

0 

-1 
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The second major group, in which 4 participants have significant positive 
loadings for Factor B, called themselves “something’s wrong” (Table 7). This 
colloquial name comes from that they have similarities in negative aspects of 
companies and universities. They appear to have features in common with 
“working with the outside world”, but they are slow to get work done, and even 
when working, they rely on social institutions for support. Indeed they analyzed 
their ineffective workability from the fact that this team could not wrap up 
discussion due to under-utilization of workshop tools such as sticky notes and 
large paper. 


Table 7. Factor B - "Something's wrong" group 




Factor Scores 

# 

Statement 

A 

B 

c 

8 

Researchers should think about how their research area can address social 

issues 

5 

3* 

0 

25 

Scientists should communicate not only the fun of their own expertise but 
also that of science in general 

0 

3* 

-2 

22 

Students should study their fields of interest in the first place 

2 

3 

-3 

21 

Students should organize Science Cafe and community engagement as 
best they can 

-1 

2 * 

-2 

16 

Universities should offer classes for the society 

-2* 

2 * 

-4 

33 

Researchers should pass down the fun of science for generations 

-1 

1 

-1 

11 

The government should provide support for the employment for a new 
field 

-2 

1* 

-3 

13 

Academic societies should disseminate more information to the public 

1 

- 2 * 

1 

10 

University research administrators should generate ideas of research 
application which researchers may not be able to think of on their own 

0 

- 2 * 

4 

9 

Off-campus networkers should promote research 

1 

-3* 

0 

34 

Graduate schools should provide university-wide career education 

2 

-3* 

0 

5 

Companies should exchange more information with universities and 
research institutes 

0 

-3* 

3 

24 

University should teach undergraduate students about social action 

2 

_4* 

-1 
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The last group comprises of 2 participants with significant positive loadings 
for Factor C (Table 8) and two other participants (Participant No. 15 and 16 in 
Table 2). As they explain, the group name “radical postgrads” means it is typical 
of postgraduates that everyone has radically different pointed remarks. Despite 
the relatively high factor score, they all disagree with “companies should support 
research for non profit” [#4] in the sense that the support from companies is just 
a part of their corporate social responsibility (CSR) activities and not so realistic 
for us to expect. There are pros and cons for supporting young researchers but 
they would normally expect “social supports” of researchers to be socially 
contributive. 


Table 8. Factor C - "Radical postgrads" group 




Factor Scores 

# 

Statement 

A 

B 

C 

6 

Companies should appreciate research career in recruiting 

-3 

-2 

5* 

10 

University research administrators should generate ideas of research 
application which researchers may not be able to think of on their own 

0 

-2 

4 * 

28 

The government should enhance international competitiveness by 
supporting gifted young researchers 

1 

1 

4 * 

5 

Companies should exchange more information with universities and 
research institutes 

0 

-3 

3* 

4 

Companies should support research for non profit 

-4 

-5 

2 * 

31 

Public sectors should assure basic management expense at the same level 
as before the transformation of incorporated administrative institutions 

-4 

-4 

1 * 

37 

Researchers should communicate with researchers working in different 
disciplines 

3 

4 

1 * 

8 

Researchers should think about how their research area can address social 

issues 

5 

3 

0 * 

35 

Individual scientists should disseminate current issues and limitations of 
their discipline 

4 

5 

0 * 

24 

University should teach undergraduate students on social action 

2 

-4 

- 1 * 

25 

Scientists should communicate not only the fun of their own expertise but 
also that of science in general 

0 

3 

-2 

22 

Students should study their fields of interest in the first place 

2 

3 

-3* 

16 

Universities should offer classes for the society 

-2 

2 

_ 4 * 

27 

Research supervisors should contrive ways to instruct researchers 

-1 

0 

-5* 


Methodological Reflection 


Q methodology has been recognized as a hybrid of qualitative and 
quantitative methods, often being called ‘para-quantitative approach’ (Capdevila 
& Stainton Rogers, 2000) or ‘qualiquantology’ (Stenner & Stainton Rogers, 2004) 
because it does not follow the hypo-deductive format that quantitative methods 
normally use (Watts & Stenner, 2005). Despite this hybridity, a growing number 
of studies have attempted to further combine Q methodology with other 
quantitative or qualitative methods. Past attempts often put more attention on 
interviews, questionnaires and surveys to increase the validity and effectiveness 
of a Q methodological study (Rutherford, Gibeau, Clark, & Chamberlain, 2009; 
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Danielson, 2009; Gallagher & Porock, 2010; Franz, Worrell & Vogele, 2013; Kim 
& Lee, 2015). 

As the above case study illustrates, person samples grouped (according to 
the Q factor analysis) are positioned with reasonable conjunctions, 
concentrations and partitions on a two-dimensional distribution space (according 
to the R factor analysis). In other words, the two distinct factor-analytic methods 
achieved comparable results and validated one another by methodological 
triangulation. A recent study demonstrates strong similarities between Q and R 
results under identical sampling conditions (Thompson et al., 2012), but the 
hybridization in Q mapping supports the basic idea that the “[s]elf is not a 
categorical construct in Q, rather it is thoroughly contextual, discursive, and 
social. It is formative, emergent, and contingent, an empirical abstraction prone 
to elaboration and understanding rather than reduction” (Goldman, 1999, p. 
592). This suggests that emergence and interaction between participants in Q 
workshop can be legitimate in Q methodology. Q workshop can be a powerful 
candidate of ‘opening-up’ elicitation and deliberation techniques to employ 
pluralistic discourse in participatory appraisal, exploring systematic divergences 
of perspective (Stirling, 2008; Stirling et al., 2007). When Stephenson proposes 
that the theory of communication ignores facts and concerns itself only with 
meanings (Stephenson, 1978), Q workshop is keen to follow the basic tenet of the 
theory in that participants can facilitate mutual communication of their 
understandings of Q sort, Q map and Q factor arrays. This heuristic employs a 
process of abductive reasoning via intensive procedures to examine substantial 
relations of connections in causal groups (Danermark, Ekstrom, Jakobsen, & 
Karlsson, 2002; Watts and Stenner, 2012). Q workshop successfully connects 
individual learning with collective learning by constructing Q sorts as individual 
mental models, which are then shared by Q factor array and by illustrating the 
difference of shared mental models on the Q map. In fact, according to follow-up 
comments from the participants after the above case event, 5 out of 10 students 
positively referred to the Q workshop exercise as a surprising finding of their 
own perspective in relation to others. 

Q methodology is closely related to narrative analysis in illustrating “the 
particular combinations or configurations of themes which are preferred by the 
participant group” (Watts and Stenner 2005, p.70, emphases in original). Like 
concept mapping, Q methodology uses predetermined structures and a rather 
strict set of rules for creating these visual representations and may provide 
individuals with a lower degree of freedom to express their own concepts 
(McNeil, 2015). Q sorting as an explicit simplified, one-dimensional mental 
model (de Haas & Algera, 2002) appears to further reduce the degree of freedom. 
Under the circumstances, participants may have to generate their own 
narratives about the given issue whilst establishing a consistent rank order 
criterion. Narratives in the deliberation and discussion of the results of Q 
mapping between participants are worth investigating for further analysis. 
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Conclusion 

Q workshop is not just recognized as a tool for the hybridity of qualitative 
and quantitative methods as a derivative of Q methodology, but rather as a 
legitimately useful tool to explore the plurality of perspectives in a systematic 
and deliberative way by reflecting on the formative, emergent, and contingent 
aspects in Q. The present study takes an empirical case on STS education for 
postgraduate students, from which we observe that the use of R factor analysis 
in conjunction with Q methodology can play a heuristic and abductive role in 
providing independent illumination of the distinguishable perspectives, 
suggesting a schematic two-dimensional basis for resolving the key differences. 

Over the last five years we have been confident of the practical utility of Q 
workshop by applying it to a number of cases, including technology assessment 
for future living, science communication for graduate education, technology 
foresight and strategy for a trading company management, and dialogue for 
innovation by forging university-industry links. We recently developed a 
dedicated Excel macro software for Q mapping that runs PQMethod in the 
background and reflects its results on the Excel sheets. Further studies would 
develop methods for evaluation of individual and collective learning at Q 
workshop by focusing on the change of mental models, or conceptual change 
(Duit & Treagust, 2003; Treagust & Duit, 2008; Chi, 2008). The evaluation can 
be performed through construction interactions among group members (Miyake, 
1986, 2008; Shirouzu, 2010), or by measuring the difference of Q maps before 
and after group discussions during the course of Q workshop. 

The two-dimensional map of perspectives generated in this study might be 
usefully applied in other cases and more general contexts — even without the 
employment of the Q-sort technique. A further elaboration of such quantification 
and graphical visualization techniques, by which individual actors can 
understand the partitioning and relative positioning of perspectives (including 
their own), should be able to provide us with more accountable, persuasive and 
reflexive measures. 
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